Authors of R packages to support Apache Spark, TensorFlow and MLflow.
The multiverse team focuses on bringing relevant machine learning technologies to R users to empower and simplify data science workflows.
“Apache Spark™ is a unified analytics engine for large-scale data processing.”
Information grows at exponential rates.
We see Spark supporting multiple projects: TensorFlow, MLFlow, Tuning, etc.
The tidyverse is an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures.
In an ideal world, all R packages work with Spark, like magic. Such is the case for dplyr and sparklyr.
library(sparklyr)
library(nycflights13)
sc <- spark_connect(master = "local|yarn|mesos|spark|livy")
flights <- copy_to(sc, flights)From launch to sparklyr 1.0.
Aspirational direction beyond 2020.